This section is intended to introduce the reader to various aspects of the art that may be related to various aspects of the present invention. The following discussion is intended to provide information to facilitate a better understanding of the present invention. Accordingly, it should be understood that statements in the following discussion are to be read in this light, and not as admissions of prior art.
The problem of detection of a human or agent's properties, behaviors, and intents has a long history. Much of the work has involved the acquisition of data by non-contact means. Most commonly, visual images have been acquired from various types of cameras and those images have been processed to infer salient information. While most non-contact data acquisition has been in the visible or near-visible part of the spectrum, other non-contact techniques to acquire observations have been used, including ones based on acoustic and infrared ranging sensing.
Use of data from direct contact with a sensing surface has also been exploited. These include occlusion systems, where the object is sensed because it blocks electromagnetic radiation in parts of the spectrum. There has also been prior work on extracting features from floor surface image data, including identification of objects (U.S. Pat. No. 8,138,882), imaging of footsteps and parts of people, and from local surface pressure measurements to characterize gait (U.S. Pat. No. 5,952,585), and to identify people. There does also exist work on tracking people using floor-sensed data.
This direct contact sensing work has used several sensing techniques. Branzel et al. (2013) describe a system that uses a camera mounted beneath a floor to image the contact of people and other objects with the floor. The system processes the visual images of the pressure contact of the objects rather than measurements of the pressure imparted by the object. Connected component analysis is applied to the image pixels to identify and track continuous areas of contact. Image features, such as image moments and shape descriptors were calculated and used as inputs to a trained feed forward neural network to assign a probability the image area matched one of seven object labels, e.g. hand, shoe, previously stored in a database. A set of heuristic association rules are used to identify particular location configurations of identified pressure image parts as being one of five poses: standing, kneeling, sitting on the floor, sitting on cube seat or sofa, and lying on a sofa.
Person tracking with floor-sensed data from pressure-activated switches or capacitive sensors Bibliography entry uses only the successive activation of the on-off pressure sensors for tracking by sensing contact. These systems do use other features of a person's locomotion, including gait parameters, or distinguish features of the path, such as turning characteristics, or actions along the path. Pressure is not sensed directly, although certain events, such as something striking the surface can be inferred.
Bibliography entry shows how sensed movement trajectories using a capacitive sensing system can be combined with heuristic rules to distinguish ‘normal’ behavior from unusual trajectories. For example, footsteps that begin at a window entrance may be indicative of a break-in in a home with a floor sensing system.
U.S. Pat. No. 8,138,882 discloses a system to identify an object on a floor by matching the detected contact shape with a shape profile stored in a database and then performing an action if some threshold values are exceeded. It describes applications in securing a premises, for example, detecting whether a person is authorized to be in a location, or that a child has entered the premises. The system can then take some action, for example, to alter lighting or notify the police or a caregiver.
U.S. Pat. No. 5,952,585 discloses a pressure sensing array apparatus that uses a plurality of current driven electrode pressure sensors to measure properties of footsteps and gait. Commercial versions, e.g. the Gaitrite system by CIR Systems, are used by clinicians to measure the gait parameters of a single person in a defined protocol of walking to support physical therapy after injuries, make analysis of gait to improve athletic performance, and to monitor the progression of certain diseases.
Although there has been work using surface contact images and pressure data to identify objects, parts of people, animals and agents, it is valuable to be able to infer specifically the classification, actions, behaviors, intents, and other properties of humans and agents in a general way and automatically. In the case of humans, such other properties include age range, gender, whether or not they have some level of knowledge, whether they are in a state of making a decision, whether they are searching for an object or information, whether they are having difficulty performing a task, and so on. Previous work on use of contact images and localized pressure data has focused only on the extraction of physical properties and their association with features in the pressure profile, or over time, for example in tracking physical location. This invention relates to extracting both physical properties of objects and associating mental and other states and properties of a human or agent, including classifications such as activity states, gender, and age, which are not immediately discernible in the pressure image and collections of pressure images.
Gait and footsteps have been studied to extract behavioral biometrics Bibliography entry. Gait has usually been studied via visual recognition systems and investigated for use in fields such as surveillance, medical applications, design and selection of sports shoes, and analysis of athletic motions and performance.
One specific and valuable problem is to measure the changing physical and mental capacities of the elderly. Heretofore, this has been performed mostly by expert human judgment based on interviewing the person, making physical measurements, and assessing the results of standard physical or mental performance tests. Such assessments can be used to assist the person to anticipate deteriorating abilities, adjust activities, and otherwise ameliorate the person's situation.
An acute problem is the prediction of a propensity to fall. Falls amongst the elderly are a significant cause of morbidity and mortality. Age-related reduction of physical capacities expressed in postural balance and gait function have been consistently linked with falls. Therefore, various laboratory and clinical tests of balance and gait have been used in attempts to predict the risk of falling.
Current practice is for a professional to conduct several tests of the person's balance and gait in a controlled setting, usually a doctor's office or clinic. Typical tests include timing the patient to rise from a seated position and begin walking (TUG test), and making gait measurements using systems, such as one available from Zenometrics LLC. Using this data, the expert makes a judgment about the person's physical capacity and their propensity to fall. One problem is that these judgments are somewhat subjective and experts may vary in their judgment given the same data and/or observations. Further, because the testing procedure takes place in a clinical setting, there are significant impediments to monitoring a patient as their physical capacities decline with age. These include cost of each visit and test, the availability of experts, and the continuity, availability, and consistency of records over time. Further, there are benefits to making gait observations during a person's natural activities because research shows that gait parameters measured in clinical settings are significantly different than those measured when a person is outside the clinical setting Bibliography entry. In addition, there is benefit to frequent observations of a person's gait.
It is desirable to have an automated procedure that predicts the propensity to fall because it can be used constantly and because it can be used by non-experts, including the elderly person and those with the responsibility of care giving. Another important benefit of an automated procedure is its ability to facilitate objective decision-making about future risks that can result in more optimal cost-benefit choices about the elderly person's living situation, for example, whether continued independent living is significantly more risky than before.
The problem of making an objective automated solution requires a means to automatically collect the observations and providing a computable representation that can be used to produce automatic objective judgments of the risk of falling that correlate well with human-based assessments. To achieve broad application of the solution in non-clinical settings it is desirable to be able to collect observations and analyze normal walking activities.
Gait characteristics have been correlated with human cognitive capacities and states, both in terms of current activities and future capacities. Certain pathological conditions are believed to affect brain function in ways that affect gait, for example by impacting motor control or split attention capacities Bibliography entry. Gait analysis and physiological tests such as rising from a seated position and beginning to walk (TUG test) can also be used to detect and predict cognitive declines of an individual. Greene describes such a system in EP2688006 A2 where inertial sensors attached to a person are used to detect gait characteristics and the inertial data can be used by a classifier to predict the future cognitive capacity of the person based on changes in the inertial data compared to a baseline.
One cause of work-related musculoskeletal disorder injuries for assembly line workers is repetitive actions where the worker assumes an awkward pose. Such injuries are estimated to cost many billions of dollars annually in compensation costs, lost wages, and lost production. The automotive industry has been a leader in this area because of significant costs due to injuries, legal action by unions, state worker compensation boards, and the insurance industry. Recognized benefits of good ergonomic design include increased factory efficiency and product quality.
Pressure patterns have been used to identify people Bibliography entry using the trajectory of the center of pressure in gait and pressure patterns of the individual footsteps. While high identification rates can be achieved in controlled settings, it is desirable to improve identification rates. Additional independent features from pressure measurements at different time scales, for example, gait parameters and locomotion behaviors, can improve classification and identification rates and reduce false positives.
Locomotion is under cognitive control and velocity patterns allow inference of locomotion goals, for example people orient themselves in ways that reflect their visual attention Bibliography entry. Visual data has been analyzed and used to automatically classify individual activity and behavior, for example as shown by Bibliography entry, to automatically identify anomalous behavior, for example as disclosed in U.S. Pat. No. 8,494,222, and to identify social relationships between people, for example as disclosed in U.S. Pat. No. 7,953,690.
Visual image-based systems have various challenges and limitations, including acquisition and simultaneous tracking of multiple objects at differing distances and with changing scale. Objects can be partially or fully-occluded and salient visual features may be cloaked. These types of limitations contribute to uncertainty in recognition and identification of objects and their properties, as well as to uncertainties in any further analysis of the data or taking action based on the data. Such uncertainties and other difficulties limit the utility of such systems in many settings, including automated security applications. They also make it difficult to extend such systems to provide new types of utility, for example attribution of behaviors to people that can be interpreted in specific contexts, such as retail shopping activity and intent. Useful object properties such as weight and detailed gait and locomotion parameters are hard to extract in uncontrolled settings. Visual systems also tend to have demanding processing requirements for large scale multi-object settings because of the need to process the images to perform basic object recognition and discern other structure in the image.
The invention makes classifications of objects and inferences about the properties and behaviors of objects based on direct measurement of the object's contact with a surface. In this way the invention solves many of the challenges and limitations of applying established methodologies and inference techniques to visual data in order to make object classifications and inferences with higher quality and better system performance.